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ABSTRACT 



The U.S. Census Bureau conducts the Schools and Staffing 
Survey (SASS) for the National Center for Education Statistics (NCES) . The 
SASS is a set of integrated self -administered surveys designed to obtain 
national information on all aspects of the school system. The Teacher Listing 
Form (TLF) is the first in the SASS series. It is sent to the principal early 
in the school year to list all teachers at the school. These results are used 
to select teachers from the school to participate in the SASS. In 1994, the 
Teacher List Validity Study uncovered some problems with the TLF. This 
document reports on cognitive research conducted to find out why respondents 
made the mistakes they did so that the TLF could be improved. A cognitive 
interview was held with one person from each of 19 sampled schools in 
Pennsylvania, Maryland, and Virginia (usually the principal) . A combination 
of cognitive techniques, including think-aloud protocols, provided data. The 
interviews revealed that the TLF needed to be reorganized so that important 
conceptual information was not interrupted by unrelated information. Some 
instructions needed to be rearranged for clarity. Better definitions of what 
to include or exclude were required. Respondents had difficulty in reporting 
the subjects teachers were qualified to teach, so a decision rule was 
designed to help them choose a subject. Breaking the data into sections that 
correspond more closely to the real-world organization of schools is also 
recommended. An appendix contains the TLF. (Contains two tables.) (SLD) 
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Foreword 



Each year a large number of written documents are generated by NCES staff and 
individuals commissioned by NCES which provide preliminary analyses of survey results and 
address technical, methodological, and evaluation issues. Even though they are not formally 
published, these documents reflect a tremendous amount of unique expertise, knowledge, and 
experience. 

The Working Paper Series was created in order to preserve the information contained 
in these documents and to promote the sharing of valuable work experience and knowledge. 
However, these documents were prepared under different formats and did not undergo vigorous 
NCES publication review and editing prior to their inclusion in the series. Consequently, we 
encourage users of the series to consult the individual authors for citations. 

To receive information about submitting manuscripts or obtaining copies of the series, 
please contact Suellen Mauchamer at (202) 219-1828 or U.S. Department of Education, Office 
of Educational Research and Improvement, National Center for Education Statistics, 555 New 
Jersey Ave., N.W., Room 400, Washington, D.C. 20208-5652. 
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Acting Associate Commissioner 
Statistical Standards and 

Methodology Division 



Samuel S. Peng 
Statistical Services and 

Methodological Research 



in 




5 



Cognitive Research on the Teacher Listing Form 

for the 

Schools and Staffing Survey 



Cleo R. Jenkins and Dawn Von Thum 
Center for Survey Methods Research 
U.S. Bureau of the Census 



February 1996 



Table of Contents 



Foreword iii 

Table of Contents v 

I. Executive Summary 1 

II. Introduction 2 

III. Methodology: Cognitive Interviews 2 

IV. Summary and Analysis 3 

Cover Page and Instruction 2 at the Top of Page 4 3 

Question 1— Only PK, K, and PS? 4 

Question 2— School District? 5 

Question 3— Administrative Unit? 5 

If "Yes" Instruction 5 

Question 4--Name? ; 5 

Instruction 1— School Level and Subject Taught Instructions 

at the Bottom of Page 3 6 

Instruction 2— Teacher Listing Instruction at the Top of Page 4 10 

Include- Exclude List 10 

The Table, Beginning on Page 5 13 

Month and Day School Closes 15 

Boxes 1 and 2 15 

Columns (a) and (b)— Line number and Name of Teacher? 15 

Column (c)— Race/Ethnicity? 16 

Column (d)— New Teachers? 16 

Column (e)~ Bilingual/ESL/ESOL? 16 

V. Recommendations 17 

Cover Page and Screening Questions 17 

Listing Instructions 19 

Month and Day School Closes 25 

Columns (a) and (b)— Line number and Name of Teacher? 26 

Column (c)— Race/Ethnicity? 26 

Column (d)— New Teachers? 26 

Column (f) through (o) 26 

VI. Conclusion 26 



Appendix Teacher Listing Form for Public and Private Schools, 
Schools and Staffing Survey 1993-94 




( 



v 



Cognitive Research on the Teacher Listing Form 

for the 

Schools and Staffing Survey 
by 

Cleo R. Jenkins and Dawn Von Thurn 
Center for Survey Methods Research 

I. Executive Summary 



The cognitive interviews revealed that the Teacher Listing Form needs to 
be reorganized so that important conceptual information is NOT interrupted by 
unrelated information, which is both distracting and potentially harmful, and 
so that it makes more sense, both verbally and non-verbal ly. 

The very important note from the cover page about what to do if the 
school’s grade range differs from the one printed on the cover needs to be 
made more prominent. Also, the screener questions (items 1, 2, and 3) need to 
be reworded. - 

What is now Instruction 1 (column reporting) and Instruction 2 (the 
list) should be reversed. This is because, operationally, respondents need to 
carry out Instruction 2 first: they need to determine who to list. After 

they have determined who to list, then they need to determine how they should 
be reported in columns (f) General Elementary through (o) Other. 

We re-designed the include/exclude 1 ist so that it mentions all of the 
categories in the table, plus it better defines the more troublesome areas: 
the "special education," "general elementary," and the "other" category. 

Also, we tried a different approach to listing the "other" teaching staff. In 
the first part of the list (Include on the List), we try to get respondents to 
think about and categorize what even they have little difficulty recognizing 
as their teachers. (For the most part, their difficulties with these teachers 
were with how to categorize them according to our definitions, which is why we 
tried to provide more comprehensive definitions.) After this, we come along 
and remind respondents to include other teaching staff they may not have 
originally thought about, the teaching principals, etc. Finally, since many 
respondents had difficulty determining whether or not to report librarians and 
speech therapists, we included specific examples regarding them. 

The cognitive interviews revealed that respondents either reported all 
of the subjects teachers were equally qualified to teach, or they agonized 
over making this decision. As it is now, we tell respondents to choose the 
subject the teacher is most qualified to teach. One would not expect this to 
be difficult, but it is. One option, and the one we are recommending at this 
point, is to give respondents a decision rule they may be able to abide by: 
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arbitrarily choose one of the subjects over the other(s). 

Last but not least, we strongly recommend that the questionnaire be 
redesigned to ask for information in a way that more closely resembles reality 
for respondents. As it is now, middle or combination schools have a lot of 
difficulty providing information in the format requested. It would be much 
better for them if we could break the data into three sections— Elementary School 
(K-5), Middle School (6-8), and Secondary School (9-12) — and collapse the data 
later to fit our needs. 



II. Introduction 



The Census Bureau conducts the Schools and Staffing Survey (SASS) for 
the National Center for Education Statistics (NCES) . SASS is composed of a 
set of integrated self-administered surveys designed to obtain national 
information on all aspects of the school system. The survey was most recently 
conducted in 1993-94, and is currently scheduled to be conducted every five 
years. 

The Teacher Listing Form (TLF) is the first in the SASS series. It is 
sent to the principal early in the school year asking for a listing of all the 
teachers at the school. It is used to select teachers from the' school to 
participate in the SASS. In 1994, a study, known as the Teacher List Validity 
Study (TLVS), was undertaken by the Demographic Statistical Methods Division 
to evaluate the quality of the data reported on the TLF. This study exposed 
problems with the TLF (Waite, 1994). For instance, it showed that public 
schools most often erroneously listed librarians and speech therapists as 
teachers. 

Although the TLVS attempted to find out why the schools excluded certain 
teachers and included persons that should not have been included, 
unfortunately the study did not obtain very insightful reasons. Most of the 
respondents simply said that they "forgot about that person" or "I thought 
this person should/shouldn’t be included." 

The cognitive research we undertook attempted to uncover with greater 
understanding why respondents made the kinds of errors that they did so that 
we could make informed recommendations for correcting these errors in future 
TLFs. 



III. Methodology: Cognitive Interviews 



We conducted cognitive interviews with one person from the school. Most 
of the respondents were principals. The remainder were those who would 
ordinarily be responsible for filling out the form: either the administrator, 

superintendent, business manager, or administrative assistants. We used a 
combination of cognitive techniques — including the concurrent think-aloud 
technique, the use of paraphrasing, and retrospective interviewing. 

Respondents were given the Teacher Listing Form to complete. We asked them to 
read aloud as they read through the form and to think aloud as they answered 
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the questions. 

We traveled to schools in four counties: York County, Pennsylvania; 

Frederick County, Maryland; and Spotsylvania and Richmond Counties, Virginia. 
We conducted nineteen interviews, distributed by school type and size (see 
Tables 1 and 2). The interviews took from 1/2 hour to 1-1/2 hours, depending 
on the size of the school. With the respondent’s permission, the interviews 
were tape-recorded and a summary of each was written. 



IV. Summary and Analysis 



Cover Page and Instruction 2 at the Top of Page 4 

A number of respondents either skimmed the cover page briefly or 
overlooked it all together. In addition, the grade range printed on the cover 
was incorrect a number of times. Although an instruction at the bottom of the 
page directed respondents to call the Census Bureau if their grade range 
differed, none of them did. Of course, an interviewer from the Census Bureau 
was present. However, at least one respondent said that he was certain he 
would not bother to call the Census Bureau under actual conditions. 

When we reached instruction 2 on page 4, it was evident that ignoring the 
grade range differences could lead to misreporting problems. Unless directed 
to specifically omit someone in the list on page 4 that follows instruction 2, 
the vast majority of respondents simply took instruction 2 to be asking them 
to list the teachers who taught at their school. The good news is that this 
was the correct interpretation when our definition of the school was actually 
incorrect. This occurred, for instance, when we defined the school as K-6, 
but the school had just recently expanded to include 7 and 8. The bad news is 
that the same situation will invite problems when we define the school 
differently than the respondent, and we really want information according to 
our definition, not theirs. This may happen in the mid-West, for instance, 
when we define a small school as two schools — an elementary school (grades K- . 
6) and a secondary school (grades 7-12)— but the school defines itself as one 
school (grades K-12). 

There are two problems with the instruction to call the Census Bureau if 
the grade range differs: 

The first problem is that respondents who skim or overlook the 
cover page never see this instruction. Furthermore, although the 
instruction at the top of page 4 directs respondents to the grade 
range listed on the cover page, it never directs respondents to 
read the accompanying instruction at the bottom of the page. 

The other problem is that even when respondents do perceive the 
instruction, they do not understand its importance (i.e., the 
implications of what it means if their grade range differs), so 
they tend to ignore it. 
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There was, however,, one respondent who actually read and was confused by 
instruction 2. This respondent acknowledged that the teacher listing should 
be of teachers within the grade range shown on the front cover, which amounted 
to sixth through eighth grade in her case. She recognized this as important 
because she had one disabled class that was ungraded in which the kids aged 
upwards to 19 years old. She was not certain if she should report this 
teacher or not. On the one hand, she decided that she should report the 
teacher because the teacher was at her school. On the other hand, she pointed 
out that some of the students were 19 years old and it was not considered a 
grade, but a program. She decided to postpone making the decision until she 
needed to, but as she became engrossed in filling out the form, she forgot 
about this situation. After being probed about this class at the end of the 
interview, she included the teacher. It is unclear if this "teacher" should 
be reported or not. 

Question l~0nly PK, K, and PS? 

This was the most misunderstood question on the form. In many cases, 
respondents answered the question incorrectly. In some cases, they answered 
the question correctly, but it was highly evident that they had difficulty 
processing it. Their reading slowed down substantially and they re-read the 
question, at least once and sometimes several times, before finally being able 
to understand it. In a number of cases, respondents answered the question 
correctly, although they gave every indication of misunderstanding it. 

One very serious problem with this question is that it asks for the 
opposite of what respondents expect. Rather than asking if the school teaches 
elementary or secondary students, which is the most common situation for a 
school, the question asks if the school teaches only prekindergarten, 
kindergarten or postsecondary students, which is a much more rare event in 
comparison. One respondent astutely pointed out our mistake when he said that 
people expect to answer "yes" to questions, and this question and the ones 
that follow are framed such that the researchers expect a "no" response. 
Confounding this situation is the fact that much of this question is dependent 
on the word "only." If respondents accidently overlooked this word and keyed 
in on the words "prekindergarten" or "kindergarten," they were likely to 
interpret this question as asking if they had prekindergarten or kindergarten. 
Of course, the answer was often "yes." 

A number of respondents had real difficulties understanding what we meant 
by "postsecondary" students. Surprisingly, respondents sometimes thought 
"post" meant "pre," as in prior to elementary school. One respondent 
correctly understood that it meant "beyond," but at first she mistakenly 
thought it meant "beyond" pre-kindergarten or kindergarten. (Eventually, she 
realized it meant beyond high school.) 

One explanation for this, which is similar to the explanation given above 
for the question itself, is that in keeping with the norms of conversation, 
respondents expect us to be asking them questions about their situations. If, 
for example, they are an elementary school, it may not easily or quickly occur 
to them that we might be asking for information outside of their realm — that 
is, for information about students beyond high school. Instead, they see the 
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question in light of their situation. In this case, "postsecondary" might 
reasonably be interpreted as "beyond kindergarten." 

Question 2 — School District? 

This question did not suffer from as many problems as the last one, but 
still respondents had difficulty with it. For instance, a public elementary 
respondent said that she had trouble understanding the article "a." She said 
that if it had said "IN a school district," she could have answered this 
question easily. The respondent was equipped with the proper knowledge to 
answer this question correctly, but she seemed to be having trouble 
understanding its intention. Why, she wondered, would we be asking her if the 
institution named on the cover page was a school district rather than IN a 
school district? Other respondents wondered this as well. One reason 
respondents were confused by this question is because we have printed the 
school s name and grade range on the cover page, giving them the impression 
that we already know it is a school. Why then, they think, are we asking this 
question. 

In addition, a couple of private school respondents demonstrated some 
difficulties with this question, although this did not impede them from 
answering the question correctly. For instance, one respondent did not know 
what a school district was. However, he assumed that since he did not know 
what it was, he must not be one, which resulted in his correctly answering 
"no" to this question. He also said that the question bothered him because it 
assumed that the school was within a school district, which was a faulty 
assumption in his case. Another private school respondent pointed out that 
private schools do not have school districts, suggesting that this question 
did not apply to him. Still he went ahead and correctly marked "no." 

Question 3 — Administrative Unit? 

Quite a few respondents had difficulty with this question, these 
respondents often overlooked the parenthetical phrase (and not a school unit), 
and this seemed to affect their interpretation. For instance, one respondent' 
who overlooked this phrase responded that yes, they were part of the County 
Public School System. 

If "Yes" Instruction 

A number of respondents overlooked this instruction, which in this case 
was advantageous, because they had incorrectly answered "yes" to one or more 
of the previous questions, but under other circumstances might be 
disadvantageous in that respondents are likely to miss this instruction when 
it truly does apply. One respondent, however, used this instruction as a clue 
that he had mistakenly answered "yes" to item 1. After learning that he 
should call the Census Bureau if he answered "yes" to any of the previous 
questions, he went back to item 1 and re-read it. He finally understood it to 
be asking if he ONLY had prekindergarten rather than if he had 
prekindergarten, and correctly changed his answer to "no." 
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Quite a few respondents had difficulty with the layout of this question. 
They had difficulty determining if the first line was supposed to be used for 
their "name" or "title." Also, one respondent wondered why we were asking for 
this information, another wondered if he should print, and a couple wondered 
if we were asking for the school’s phone numbers. 

Instruction 1 — School Level and Subject Taught Instructions at the Bottom of 
Page 3 

Most respondents read through, or at least skimmed the instructions 
before completing the form, and referred back to them when completing the 
form, indicating that the instructions were necessary for completing the table 
later on. However, there were two major problems with the instructions. 

One, respondents perceived the instructions as burdensome, which in turn 
meant that they perceived the task of filling out the questionnaire as more 
difficult than it really was. One respondent, who presided over an elementary 
school, was clearly frustrated that he needed to wade through seven 
instructions when only two of them applied to his school. The other five 
pertained to secondary schools. He could not understand why this was 
necessary. After all, we had clearly demonstrated that we knew his was an 
elementary school by labeling the front of the questionnaire with K-5. 

Two, respondents often had difficulty perceiving, understanding, or 
agreeing to carry out the instructions, as noted in the remainder of this 
section. In response to the first instruction "If you need assistance, please 
call the Census Bureau..." a few respondents said that they would not bother, 
even if they needed help with the form, while a couple of respondents skipped 
over this instruction entirely. Even more respondents overlooked the "Please 
return the form within 2 weeks" instruction. These respondents were drawn to 
the bolder question "How are columns (f)-(o) completed?" just below the 
instructions. Not only is the question made more prominent than either 
instruction because it is bolder and bigger, but it is numbered "1," all of 
which are non-verbal cues attracting the respondent’s attention. It is worth 
noting that the part of the instruction telling respondents to return the form 
in the enclosed envelope really is useless at the beginning of the survey. It 
may be important to tell respondents to complete the form in two weeks up 
front, but they should be told how to return the form at the end of the survey 
when they actually need to act on this information. Otherwise we are 
needlessly asking them to store the instruction in memory, or we are asking 
them to hunt for this instruction later on, and from the respondent’s 
perspective, this instruction is then buried among many instructions. One of 
the respondents who overlooked this instruction at the beginning of the survey 
did spend time at the end hunting for something to tell her what to do next. 

Many respondents wondered what the question "How are columns (f)-(o) 
completed?" meant, since nothing on this page indicated to what "columns (f) - 
(o)" referred. One respondent seemed to sum up most respondents’ reactions to 
this question when he said, "...where in the hell are columns (f) through 
(o)?". After wondering about this, most of the respondents generally turned 
the page and found the columns. However, some never did turn the page. 
Although confused, they decided to continue reading through the instructions. 
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They probably figured that the instructions would eventually settle this 
issue, although they never do. 



One respondent misunderstood the instruction in paragraph 4: 

If a teacher teaches both elementary and secondary students at 
this school, mark (X) in the appropriate column under the level 
(Elementary or Secondary) at which the teacher teaches the greater 
number of classes. 

This respondent read this instruction before he began to fill out the 
questionnaire, but he forgot exactly what it said when he reached Teacher 8 to 
whom it applied, so he went back, found this instruction, and re-read it. 
However, he only read as far as "mark (X) in the appropriate column" and then 
he went on to read the next instruction, which gave him the impression that he 
was to mark (X)’s in both sections. 

Of all of the instructions, respondents had difficulty understanding the 
instruction in paragraph 5 the most: 

"If an elementary teacher teaches a departmental class, e.g., 
music, art, reading, math, or science, mark "x" in column (h) 
other, under elementary. Mark general elementary only for 
elementary teachers who teach in self-contained classes, i.e. , 
teach the same class of students all day or most of the day." 

Respondents’ misunderstanding of this instruction is probably not a 
simple matter of misunderstanding this instruction alone, but the result of 
(1) mixed messages coming from different sources on the questionnaire and (2) 
the fact that elementary schools are instructed to report similar kinds of 
information as secondary schools in categories that are labeled differently. 

To begin with, the instruction in the third paragraph says "If a teacher 
teaches only students in grade 6 or below, mark the appropriate subject column 
under "Elementary." This paragraph is written in parallel structure to the 
preceding paragraph when it says to "mark the appropriate subject column under 
"Elementary." The problem with this construction, however, is that there are 
no subject columns under "Elementary," at least not subject columns like there 
are under "Secondary." Only the columns "general elementary, special 
education, and other" are provided. In comparison, the secondary section 
contains what are clearly subject columns, Tike "math, science, and English." 

This is especially confusing for respondents who find themselves in the 
unenviable position of having to report under both the elementary and 
secondary sections. Since the headings of these sections are not the same, 
respondents are required to report the same kinds of information under 
different headings. For instance, a sixth grade science teacher would be 
reported in (h) other, whereas a seventh grade science teacher would NOT be 
reported in the corresponding column (o) other, but in the science column (j). 

One respondent incorrectly reported an elementary/middle school science 
teacher in the "special education" column in the elementary section. The 



ERIC 



14 



8 



respondent said that he was reporting the science teacher in this column 
because she taught more than one grade of science. She taught science to 
second, third, fourth, fifth, and sixth grade. After the respondent 
incorrectly reported the teacher, he turned back to the instructions for 
guidance. It is impossible to tell from what the respondent said if he turned 
back to the instructions because of the interviewer’s probing, but the 
interviewer suspected so. The respondent read the instruction "If an 
elementary teacher teaches a departmental class, music, science, etc." and 
discovered his mistake. He then put his mark under "other" in the elementary 
section. 

It appeared to the interviewer, although the respondent never said this, 
that the respondent had quickly ruled out the "general elementary" column, 
which left either the "special education" or the "other" category; The 
interviewer speculated that the respondent did not think that science should 
be classified as "other," especially since the respondent knew from previous 
experience that there was a "science" category on the secondary side. The 
interviewer thought that the respondent ruled out the categories "general 
elementary" and "other" and by a process of elimination decided upon the 
category "special education." 

Another respondent was also clearly influenced by the secondary columns, 
although this respondent came to the opposite conclusion as the previous one. 
This respondent began by correctly reporting two sixth-grade subject-matter 
teachers as "other" teachers on the elementary side. However, after he next 
reported a secondary teacher, the respondent became confused about how to 
report his elementary teachers, and from that point onward, he made the 
mistake of reporting what were essentially sixth-grade subject-matter teachers 
as "general elementary" teachers. The reasons for this in the respondent’s 
words were: 

1. "I’m looking back on page 3.. I don’t see anything there 
that will help me." [The respondent is referring to 
choosing between columns (f) or (h ) . ] ; 

l ‘ 

2 . "...they have elementary, special education and other. Both 
of those ladies would, in my opinion, be considered to be 
’general education’ rather than ’other.’ I would think that 
’other’ category might apply if they were a music specialist 
or some other type of specialist that worked at the sixth 
grade level, not teaching math, science, social studies, or 
language arts. That’s what I consider to be general 

el ementary. " 

It is only normal that once a respondent has reported a secondary subject 
matter teacher under one of the subject matter headings, because of what is 
known as top-down processing , they would expect to do the same in the 
elementary section (or vice versa). In general, top-down processing lends 
efficiency to our ability to interpret stimuli. It enables us to quickly 
perceive our world with less effort by placing what are usually well-founded 
expectations upon it from past experiences. However, a break-down in this 
system is likely to occur when the world deviates from our expectations, as it 
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did for the respondents above. When this occurred, different respondents 
opted for different columns for different reasons. 

Looking again at paragraph 5, one notices that the important instruction 
"mark (X) in column (h) ("Other") under Elementary" can be lost. This might 
happen because the respondents assume that the second line is nothing but 
examples, all of which appear to be separated by commas. If so, their eyes 
may drop to the capitalized verb "Mark," with the result of giving the wrong 
instruction: "Mark "General elementary." The above respondent did report 

nearly all of his sixth grade departmentalized teachers under the "General 
elementary" column. 

In addition, there is a discrepancy between the way the answer columns 
in the table are organized compared to this paragraph. The answer columns in 
the table are organized as follows: "general education," followed by "special 

education," then "other." This paragraph, however, discusses the "other" 
category first, followed by "general elementary." There was at least one 
respondent who looked for an instruction telling her how to report the bulk of 
her elementary teachers (by which she meant her general elementary teachers), 
but who could not find the appropriate instruction because it is buried at the 
end of this paragraph. Moreover, the paragraph is incomplete because it never 
mentions "special education." 

Finally, many respondents questioned what we meant by the instruction in 
paragraph 7: 



"If a secondary teacher teaches two subjects equally, mark the 
column for the subject the teacher is most qualified to teach." 

There were difficulties determining which subject the teacher was most 
qualified to teach. Respondents often indicated that teachers can have 
certificates in more than one subject, which means that the teachers were 
equally qualified to teach any of these subjects. In this case respondents 
were left without a decision-making criterion to guide them in their choice, 
and this bothered them. As a result, a few respondents marked "X"s in more 
than one column. 

These respondents took the word "qualified" to mean "certified." 

However, one respondent pointed out that "qualified" can mean something 
different from "certified." Certified implies that the teacher has more 
concentration in formal learning in certain fields. Qualified can have that 
meaning, or it can refer to whatever subject/skill the principal feels the 
teacher is best at, whether that skill stems from formal training or personal 
interest. When queried, this respondent, similar to the others, interpreted 
"qualified" to mean "what their license says." 

One private school respondent had a special problem because there was not 
an equivalent instruction about what to do if an elementary teacher taught two 
or more "subjects" equally. This respondent had a teacher that taught first 
grade all morning, but in the afternoon the teacher taught French to grades. 1, 
2, and 3, as well as religion. This respondent decided to put an "X" in the 
"general elementary" column, and a "?" in the "other" column. It truly was 
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unclear how this teacher should have been reported. 

Another private combination school had a similar kind of problem in that 
there was not an instruction about what to do if a teacher taught at both the 
elementary and secondary level equally. This respondent filled out the form 
quite well, until he reached Teacher 4, who taught both 6th and 7th grade. 

The previous teachers had been easy to categorize, since they exclusively 
taught self-contained classes in the elementary grades. But when it came to 
Teacher 4, the respondent first marked an "X" under the other column (h) in 
the elementary section, but then questioned whether this was correct. He 
seemed to be confused because Teacher 4 did not teach a self-contained class. 
He went back to the instruction to find out what to do. At first, he read 
what to do if the teacher taught students only in 7th grade or above, and 
ruled out this instruction. He then read the instruction about what to do if 
the teacher taught students only in 6th grade or below, and as with the 
previous instruction, ruled this out. Then he read what to do if the teacher 
taught both elementary and secondary and learned that the decision rule was to 
mark the level (elementary or secondary) at which the teacher teaches the 
greater number of classes. This posed a problem, however, because he said 
that the teacher taught at both levels equally. The respondent decided to 
keep the "X" under column (f ) . He then noted the fact that the instruction 
said "Mark (x) in ONLY ONE of the columns (f) through (o)." He did not agree 
with this, however, because he said: "So I either mark her down under all of 

those, or mark "other" and let you guess." After saying this, he went on to 
put an "X" in columns (i), (j), and (k) to signify the fact that this teacher 
taught "math, science, and english" to 7th graders. 

When done reading the instructions, one respondent mistakenly declared 
that she did not need to worry about "secondary," although she had 7th and 8th 
grade students at her school. This suggests that the instructions were not 
enough to change her definition for secondary from 9th grade and above to the 
form’s definition of 7th grade and above. In other cases, respondents 
recognized that the form’s definition differed from theirs, but wondered why. 

Instruction 2 — Teacher Listing Instruction at the Top of Page 4 

Discussed earlier along with the cover page. 

Include-Exclude List 

For the most part, this list was beneficial. Generally, respondents 
either fully read or at least skimmed through the list before beginning to 
answer the questionnaire, and they returned to it as necessary (although one 
respondent completely ignored the list until he was in the process of listing 
the teachers, at which point he found himself in need of guidance). There 
were, however, a number of ambiguities with the list that need to be 
corrected. 

To begin with, respondents had difficulty determining why certain kinds 
of teachers were listed (e.g., art and physical education teachers) and not 
others (e.g., foreign language teachers). This often made them question if 
they were supposed to include teachers like the foreign language teachers. 
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This problem was exacerbated by the fact that foreign language teachers are 
also not listed in the table headings, whereas some categories of teachers 
are, like English and science teachers. This meant that respondents needed to 
spend time determining that a teacher like a foreign language teacher simply 
fell under the heading of full- or part-time teacher in the Include/Exclude 
List. Then they had to determine that this kind of teacher would be reported 
under "other" in the table. 

Respondents had difficulties understanding what was meant by "a regularly 
scheduled class, " especially when it came to librarians and speech 
therapists, and thus they often erroneously included them. Since this phrase 
is clearly pivotal to determining whether many school personnel get listed, it 
needs to be made clear. According to respondents, librarians sometimes ao 
into "regularly scheduled" classes to instruct them in library skills. The 
same with speech therapists. As one respondent put it, the speech therapist 
did not teach whole classes, but sometimes she went into the regular classroom 
and taught that class. The respondent explained that the speech therapist 
worked with the speech kids within the group of the regular class. 

Respondents wondered if that counted. 

One respondent had difficulty reporting, as he put it, a person who 
coached one class during the season. The respondent reported this person as 
an "other" teacher in the secondary section, but the respondent recommended 
that the instructions be made more specific about how we wish to treat people 
who actually do not teach in the classroom. Later on, the respondent reported 
another coach. 

Finally, respondents wondered if the word "class" excluded individual 
instruction. As a, result, they wondered if the speech therapist who often 
works with individual students should be excluded. A common theme running 
through all of these situations is that respondents are wondering if 
physically being in a classroom is the defining characteristic here. 

At least a couple of respondents had difficulties with the opposite 
instructions "Include on the list persons who teach a regularly scheduled 
class but whose main assignment is:" and "Omit from the list persons who do 
not teach any regularly scheduled classes and whose main assignment is:". 

They either seemed to feel that the replication was a waste of their time 
and/or they had difficulties distinguishing between the two statements. At 
least one respondent thought that the one list said to report the librarian, 
etc., the other said not to. Distinguishing between these two statements 
requires focusing on the "who teach/do not teach" part of the statement rather 
than the "whose main assignment is" part, and some respondents had trouble 
with this. 

In addition to the overall "include" statement being confusing, the use 
of the subordinate conjunction "but" may be inappropriate. More often than 
not the survey was being conducted with the principal, who was the top person. 
(Does not the questionnaire acknowledge this by listing "Principal" first in 
the list?) The way the statement is worded essentially makes the principal 
job secondary to the teaching one. That is, "teach..., but a principal," not 
"and a principal" nor "principal and teach." The difference is subtle, but 
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probably worth noting. It may be confusing that "principal" would be 
subordinate to "teacher." 

Also, there was difficulty determining what was meant by "special 
education." One private school respondent looked for and remarked that there 
was not a definition for "special education." With probing, the respondent 
gave as an example the reading remedial specialist. With further probing, he 
said that special education might mean "special teachers who are there because 
of special education programs." He pointed out that there was a difference 
between someone who signs for a hearing impaired child and someone who is 
simply spending more time with a child who needs some special attention to 
read. He seemed to have trouble determining if we meant one of these over the 
other. He went on to explain that his special teachers may spend something 
like thirty minutes or so drilling and giving special attention in reading. 

He wondered if this was enough to qualify them as special education teachers. 
He compared this activity with the training of children who are dyslexic or 
who have a learning problem, like those in public schools. In the end, he 
decided to report the remedial reading teacher in the special education 
category. 

However, another respondent reported his learning disabilities teachers 
as "general elementary" teachers because they taught self-contained classes. 

No doubt this occurred because a definition for special education was not 
provided on the questionnaire and because the instructions specifically say 
that if a teacher teaches a self-contained class they should be reported as 
"general elementary" teachers. If "special education" teachers are supposed 
to be an exception to this rule, it is not specified. 

Still another respondent debated whether to report the teacher of gifted 
and talented students under special education. The respondent pointed out 
that the gifted and talented program is part of the exceptional program, but 
it is generally not considered special education. In the end, the respondent 
decided to report the gifted and talented teacher under "other." 

Finally, a few respondents mentioned not knowing what itinerant or 
satellite teachers were. The way in which these categories are grouped 
separately on the page suggests that these are two different kinds of 
teachers, when in fact they are varying names for the same thing. This is an 
example of the Gestalt Grouping Laws in operation, whereby respondents are 
relying upon the visual element of location to provide them with meaningful 
information. Their mistake — which of course is not really their mistake, but 
ours— stems from the fact that the visual grouping is misleading. 

There was a little confusion over "long-term substitute teachers." One 
principal did not "currently" have a long-term sub, but had just had one for a 
teacher on maternity leave and planned on having one again for another teacher 
going on maternity leave. The principal wondered if he was supposed to report 
these teachers, but decided not to. The same thing happened with another 
principal, only this principal decided to include the non-current long-term 
sub. 
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The Table, Beginning on Page 5 

When respondents get to the table on page 5, they are first asked to 
enter the month and day school will close. Then they are asked to fill out a 
table in which they are supposed to report the teachers’ names (in column b), 
race/ethnicity (in column c), total years in teaching (in column d), whether 
the teachers are bilingual (in column e), and then the grade level and subject 
taught by the teachers (in columns f through o). 

We have decided to skip over the date school closes and columns (b) 
through (e) temporarily. Instead, we begin by discussing respondents’ overall 
listing strategies. After that, we discuss the errors that arose as 
respondents tried to fill in columns (f) through (o). The reason we begin 
like this is because this information is conceptually connected to the 
instructions on pages 3 and 4, which we have just reviewed. When we are done 
with these aspects of the listing operation, we come back and talk more 
specifically about columns (b) through (e). 

Listing Strategies 

Whether working from a master schedule or from memory, respondents 
often wondered if they were supposed to list teachers alphabetically, or 
by grade level or subject. About one-third of the respondents reported 
their teachers from memory by grade level or subject. These respondents 
were relieved that there was not an instruction telling them that they 
needed to report their teachers alphabetically because it turned out that 
this information was stored in memory by grade level or subject. 

The remaining two-thirds of the respondents worked from master 
schedules. A few of these had two schedules, one arranged 
alphabetically, the other by grade level or department. The majority of 
the respondents that worked from a schedule decided to list the teachers 
alphabetically, which turned out to be less efficient than when 
respondents listed them by grade levels or departments. This is because 
the respondent had to move back and forth between columns (f) through 
(o), constantly repeating the columns and looking for the right one, 
whereas respondents reporting by departments could move more efficiently 
down the page, marking one column for awhile before moving on to the next 
one. Another advantage of working from a schedule that was arranged by 
department was that respondents could often skip over groups of people 
who were to be excluded from our list (e.g., teacher aides), rather than 
having to identify each respondent’s position and then individually 
exclude them as they went along. 

Most respondents filled out the matrix by working across one row 
at a time, then moving to the next row, etc. However, three respondents 
worked down the columns, first listing all of the teachers, then 
reporting their race/ethnicity, etc. 
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Grade Level Mismatch 

One of the largest problems with the table came from the mismatch 
between the form’s definition for elementary and secondary grade levels 
and the respondents’ definitions. The form defines an elementary school 
as containing kindergarten through sixth grade and a secondary school as 
seventh through twelfth grades. However, most of the schools in our 
study defined elementary school as kindergarten through fifth grade, 
middle school as sixth through eighth grade, and secondary school as 
ninth through twelfth grades. 

The middle schools in particular had difficulty determining how 
they should report their teachers because of the effort required to 
determine "the level at which the teacher taught the greater number of 
classes." As noted earlier* one respondent understood this instruction, 
but had a problem anyway because the teacher taught at both levels 
equally. He decided to mark more than one column. Another misunderstood 
the instruction. As a result, he put "X"s in more than one column. It 
is worth noting, however, that even when respondents understood this 
instruction, they struggled with this task. 

Misreporting of Teachers 

It is fair to say that most respondents reported the majority of 
their teachers properly. Not surprisingly, these were the full-time 
classroom teachers— those who taught grades 1 through 5 or subjects in 
the higher grades, no doubt because these are unambiguously teachers (by 
practically anyone’s definition). However, this is not to say that 
respondents did not have trouble reporting their staff. As can be 
surmised from what we have written up to this point, nearly every, if not 
all respondents, questioned whether or not they were supposed to report 
one or more of their staff. Respondents questioned whether to report the 
librarian probably more than any other person on staff, but they also 
questioned whether to report speech therapists, guidance counselors, 
extended enrichment staff, ESL teachers, foreign language teachers, 
coaches, remedial reading specialists, LD teachers, teachers of 
"ungraded" classes, junior kindergarten teachers, teachers for the 
visually impaired and the hearing impaired, volunteer teachers, and 
teachers who only taught a little bit (e.g.,_one respondent had a teacher 
who taught art a little bit of the time, but was a kindergarten assistant 
the majority of the time). 

Perhaps we can learn something from one of our respondents, who 
had a list which categorized her employees. There were four major 
categories on her list: grade-level teachers, a specialist team, a 

resource team, and other staff. The respondent did not have any 
difficulty knowing to report the grade-level teachers or, for the most 
part, to omit the other staff. It was the specialist team and resource 
teams, as well as the itinerant staff in the other category that gave her 
the most trouble. The specialist team included the phys. ed., media, and 
instrumental music staff, all of whom she reported as teachers. The 
resource team included the special education and integrated language arts 
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staff, who again she reported. The other category included the 
instructional assistants, cafeteria staff, and itinerant staff. The 
itinerant staff included the psychologist, the people personnel worker, 
and the extended enrichment person. Of these, the respondent reported 
the extended enrichment person. 

Table Headings 

A number of respondents had difficulty filling out columns 
(f) through (o) when they reached the odd-numbered pages because 
the headings were so far from where they were working. 

Month and Day School Closes 

There were quite a few problems with this item. One respondent 
overlooked it, perhaps because it is sandwiched between what are essentially 
the most important parts of the questionnaire: (1) instructions for listing 

the teachers, and (2) the area for actually listing the teachers. And to make 
matters worse it is conceptually unrelated. Although this is not the sort 
of thing respondents are likely to mention, it is distracting (and therefore, 
potentially harmful) that we physically ask this question here. 

In addition, respondents had some difficulties interpreting this 
question. One respondent interpreted day to be "day of the week," so she 
wrote in "Friday." But by far the largest problem was that respondents 
wondered if this question was asking them for the day school closed for the 
students or for the teachers, which were different times. Teachers often had 
in-service days beyond the students. Also, in one school, kindergarten ended 
a few days earlier than grades 1 through 6. Most respondents decided the 
question was referring to the grade-level students, but at least one reported 
the day school closed for teachers. 

Boxes 1 and 2 

Many respondents questioned the meaning of these boxes. At first, quite 
a few respondents were convinced that they had overlooked some instruction 
telling them how to fill out these boxes. Usually this gave them slight pause 
until they figured out that the boxes must not pertain to them. Surely, this 
is a waste of the respondent’s time. Also, some respondents never did figure 
out what the boxes pertained to, but did not know what to do with them, so 
they skipped over them. One respondent said that he would return, (presumably 
when he had more knowledge about what to do with them), but he never did. 

Columns (a) and (b) — Line number and Name of Teacher? 

A couple of respondents ignored the instruction to provide the teacher’s 
last name first, and one respondent misunderstood the line numbers to the left 
of the name column. She took them to mean grade level, so for instance, she 
reported the first grade teacher on line 1, the second grade teacher on line 
2, etc. When she was done with the grade-level teachers, she reported the 
other teachers, like the art teacher, but because of the way she had been 
using the line numbers as cues, she forgot to report the kindergarten teacher. 
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Also, one respondent could not report the last names of the Sisters who taught 
because she did not know them, and one respondent wanted to know if he should 
print the names. 

Column (c) — Race/Ethnicity? 

This item did not pose many problems with our respondents because most of 
the teachers in the areas we interviewed were white, with a few blacks and 
Hispanics. One respondent accidently wrote in "l"s rather than M 5"s to 
indicate white. She just assumed that the predominate race (white) would be 
listed first. At some point, she realized her mistake and converted the "l"s 
to " 5 " s . 

Column (d) — New Teachers? 

There were a number of problems with this item, the greatest of which was 
that a number of respondents neglected to think in terms of a teacher’s total 
years in the teaching profession "at all schools" when they first began to 
fill out the questionnaire. Again, one reason for this may be that the 
respondent’s situation came to mind first. Most respondents, however, caught 
their mistake. One respondent mentioned that state reports ask for teaching 
experience at all schools too. Perhaps this is why respondents tended to 
realize their mistakes at some point. 

In addition, there were a number of other problems with this item. For 
one, a couple of respondents put "X"s in this column rather than "N"s, more 
than likely because up till this point the instructions had been saying to 
mark "X"s. This would be another example of top-down processing at work. In 
addition, a couple of respondents had difficulty understanding the phrase "not 
counting this school year." Another had difficulty with the phrase "if a 
teacher is less than 3 years at any school." Still another had difficulty 
determining if this question was asking for state or national figures. And 
still another wondered if she should include college teaching when determining 
the teacher’s total teaching experience. One respondent had difficulty 
determining the teacher’s total teaching experience because the teacher’s 
length of teaching included interrupted service. And finally, one respondent 
was unable to answer this question for teachers who were not home-based at his 
school . 

Column (e)— Bilingual /ESL/ESOL? 

As with the race/ethnicity item, respondents did not seem to have too 
many problems understanding this item, mainly because the respondents in our 
sample did not offer these programs. However, one respondent misread the note 
at the bottom of this item. The note reads: "Do not include regular foreign 

language teachers as Bilingual, ESL, or ESOL teachers." However, this 
respondent simply read this note as "Do not include foreign language 
teachers." As a result, she erroneously omitted her foreign language teacher 
from the list. 
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V. Recommendations 

For the rest of this report, the question numbers correspond to the 
revised questions, not the ones on the current form. 

Cover Page and Screening Questions 

The questionnaire should be reorganized so that important conceptual 
information is NOT interrupted by unrelated information, which is both 
distracting and potentially harmful, and so that it makes more sense, both 
verbally and non-verbally. The questions should begin on page two rather than 
three. Rather than being double banked, they should run down the page 
vertically. 

The very important note from the cover page needs to be made more 
prominent. One way to do this is to combine this instruction with what are 
now items 1 through 3. It has been our experience that respondents tend to 
read information marked by question numbers. Also, the cognitive interviews 
showed that respondents need to understand the purpose of the screener 
questions, so we recommend the following: 

1. To assure that we have sent this form to a school, rather 
than to a school district or an administrative unit within a 
school district, please read and answer the questions below. 

2. Is the institution named on the cover page a school? 

[ ] Yes 

\ [ ] No 

s/ 

If you answered "No," it is very important that you call the 
Census Bureau toll-free at 1-800-851-2014. Otherwise, go to 3. 



Is the grade range listed on the cover page correct? 

- ~ [ ] Ves 

i [ ] No 

\u 

If you answered "No," it is very important that you call the 
Census Bureau toll-free at 1-800-851-2014. Otherwise, go to 4. 



The above questions are framed such that respondents are expected to say 
"yes," which is in keeping with their natural tendencies. Also, they are 
based on the assumption that school districts and units within school 
districts are more likely to know that they are not schools than vice versa, 
especially after being given the above introduction. 

More importantly, the original item 1, which was very hard for 
respondents to understand, has now been subsumed by the new question 3. Are 
we not right in assuming that an all -prekindergarten, -kindergarten, or - 
postsecondary school would never have been listed on the cover page as such? 
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If a respondent says "yes" in response to the new question 3, then we can 
assume they do not have ONLY prekindergarten, kindergarten, or postsecondary 
students, that they are who we thought they were. If they say "no," then we 
want to speak with them to find out if they are out-of-scope, have different 
definitions of their school than we do, or have changed since we last listed 
them. 



Finally, these questions use the same skip instruction as used on the 
Teacher Questionnaire, which we think respondents may execute relatively well 
compared to other skip instructions (However, please note: This information 

is preliminary. The final results are forthcoming in a report that evaluates 
the Teacher Questionnaire). 

It would probably be a good idea to replace the note to call the Census 
Bureau if the school’s grade range differs on the cover page with the "Note: 
Please return this form within 2 weeks in the enclosed envelope." Ordinarily, 
we would have recommended that this instruction be put at the end of the 
questionnaire, but respondents stop listing at various points along the way, 
depending upon the number of teachers at their school, so there is no clear 
end to this questionnaire. 

The question concerning the month and day school closes should become 
item 4. This question needs to include a reference to the date school is 
expected to close for students, if that is the date we want. If not, then it 
needs to include a reference for teachers. Also, it should say "date" rather 
than "day," since some respondents take "day" to mean "day of the week," like 
Monday or Tuesday. 

4. What month and date will school close for students at the 
end of the 1993-1994 school year? 

Boxes 1 and 2 should be deleted and the respondent’s name should become 
item 5. With regard to item 5, we suggest clearly specifying the information 
we are after, as well as providing a brief explanation as to why we are asking 
for this information. Most importantly, we need to make sure that respondents 
can tell exactly where they are supposed to write the information we request, 
since this was the largest problem with this item in the interviews. 

5. In case we have questions about any of your responses, please 
print your name, title, and work telephone number on the lines 
below. 

Your name: 

Title: 



Work Telephone Number: ( ) 

Area 

Code 



Also, we recommend continuing to follow the name item with the 
instruction to call the Census Bureau if respondents need assistance, similar 
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to the way it is now in the questionnaire. Respondents just got finished 
telling us their name and telephone number, so it seems like this is the 
appropriate place for us to tell them ours and to offer our assistance. 

6. If you need assistance filling out the rest of this form, please 
call the Census Bureau toll free at 1-800-851-2014. 



Listing Instructions 

What is now Instruction 1 (column reporting) and. Instructi on 2 (the list) 
should be reversed. This is because, operationally, respondents, need to carry 
out Instruction 2 first: they need to determine who to list. After they have 

determined who to list, then they need to determine how they should be 
reported in columns (f) General Elementary through (o) Other. Rather than 
starting over with the number one as the current form does, the instructions 
should just continue in the same number sequence. What is now instruction 2 
will become items 7 and 8, and what is now instruction 1 will become item 9. 
The new items 7 and 8 should be put on page 3, and the new item 9 on page 4. 

At this point, we should not tell respondents to report the teachers in 
the grade range shown on the cover page because that information may be wrong. 
So long as the screener questions work and respondents call the Census Bureau 
if their grade ranges differ from what is listed on the cover paqe, then Item 
7 should be: 

7. Please read items 8 and 9 below, then list and categorize the 

full- and part-time teachers at this school in the table on page 



A possible advantage of this reorganization is that the first set of 
instructions respondents come across will be the easier-to-process list, which 
may not turn. them off as much as the dense and detailed instructions they now 
encounter first. However, a disadvantage of this reorganization may be that 
the list, which was easy for respondents to refer to, will no longer be 
directly in front of them at the beginning of the listing process (i.e., when 
they are on page 5) the way it was when it was located on page 4. Because of 
this, we considered recommending a fold-out page, so that both sets of 
instructions could be above the table and in front of respondents, at least at 
the beginning of the listing process, and we even considered having a separate 
sheet of instructions that respondents could keep in front of them at all 
times, but we decided that both of these recommendations would create more 
problems than they solve. In the case of the fold-out, the column 
instructions will be covered once respondents turn the page and it will 
probably end up being clumsy and burdensome for respondents to handle anyway. 
And with regard to the separate sheet, respondents are likely to ignore or 
lose it. 

At this point in our thinking, we believe that one of the best ways of 
keeping respondents from overlooking or ignoring important information may be 
to first make certain that the information flows from a logical standpoint and 
then to sequentially move them through the information with the use of item 
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numbers. Therefore, in item 7 above, we direct respondents’ attention to the 
two very important sets of instructions that follow (in items 8 and 9), before 
telling them about the table on page 5. 

A revised version of the include/exclude list is presented below in item 
8. After the list, we discuss in detail the changes we made to the original 
list and why. 

8. INCLUDE ON THE LIST 

Special Education Teachers 

meaning those who teach the emotionally disturbed, mentally 
retarded, speech/language impaired, hearing impaired 
visually handicapped, orthopedical ly impaired, mildly and 
severely handicapped, learning disabled, etc. 

General Elementary Teachers 

1. . meaning those who teach in self-contained classes, 

i.e., teach the same class of students all or most of 
the day, unless they teach special education students, 
in which case see the category above 

2. include kindergarten teachers 
Math Teachers 

Science Teachers 
English/Language Arts Teachers 
Social Studies Teachers 
Vocational -Technical Education Teachers 
Other Subject-Matter Teachers 

1. meaning those who teach art, foreign languages, music, 
physical education, English as a second language, and 
any other remaining subjects 

2. include those who teach a remedial reading class 

3. include those who teach the gifted and talented 

ALSO, REMEMBER TO INCLUDE AND CATEGORIZE 

Teaching Principals, Teaching Guidance Counselors, Teaching Speech 
Therapists, Teaching Librarians, Teaching Psychologists, and 
Teaching School Nurses 

1. this includes any staff member who teaches at least 
one class per week 

2. for example: 

if librarians teach a class in informational 

science once a week, they would be included in 
the "other" category, but if they teach library 
skills once a year or if they go into other 
classes to teach library skills, they would NOT 
be included on the form 
if speech therapists teach a class in 

remedial reading once a week, they would be 
included in the "other" category, but if they 
tutor an individual child or children who need 
extra help learning to read, they would NOT be 
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included on the form 
Teachers of Ungraded Students 

1. meaning those who teach ungraded students who, at 
least in part, are comparable in age to the students 
the school was designed to serve; 

2. for example: 

if a middle school has a class that contains 

ungraded students who range in age from 12 to 19 
years old, this teacher would be included 

3. report these as "other" teachers, unless they teach 
special education students, in which case report them 
in the "special education" category 

Itinerant, Co-op, Traveling, and Satellite Teachers 

meaning those who teach at more than one school 
Current Long-Term Substitute Teachers 

meaning those who are currently filling the role of a 
regular teacher for 4 weeks or more 

OMIT FROM THE LIST 

Teacher aides 
Prekindergarten Teachers 

so long as they teach ONLY prekindergarten students 
Postsecondary and Adult Education Teachers 

so long as they teach ONLY postsecondary and adult education 
students 

Short-term Substitute Teachers 

meaning those who fill the role of a regular teacher for 
less than 4 weeks 
Student Teachers 
Day Care Aides 



Because of the space constraints we are facing here, we simply continued 
the omit list down the page after the include list, but We really suggest that 
the omit list be double banked, the way it is now in the questionnaire. 

We have designed the include/exclude list so that it will perfectly 
correspond with what we envision as the table. Since the references to full- 
and part-time teachers do not correspond with the categories provided in the 
table, we moved them. In truth, everyone falls into the full- or part-time 
categories, so we put this reference where we actually want a global 
reference: in the question itself (Q7). 

Furthermore, we suggest that the "special education" category be provided 
first, both in the list and in the table. This is because respondents are not 
likely to mistakenly report their general elementary teachers, or any of their 
other teachers for that matter, as special education teachers (as they are 
presently doing), j_f they have first read the definition for special education 
teachers. The opposite is not true, however, since the definition provided 
for general elementary teachers (teach the same class of students all or most 
of the day) can also apply to special education teachers. It is not classroom 
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structure that differentiates the two, but the populations to which they 
teach, and respondents need to be made aware of this. 

The above list is also much more comprehensive than the original list. 

It mentions all of the categories in the table, plus it better defines the 
more troublesome areas: the "special education," "general elementary," and 

the "other" category. We did not bother to elaborate on the categories 
respondents did not seem to have much problem defining for themselves, like 
math, science, social studies, and English. 

Finally, we tried a different approach to listing the "other" teaching 
staff. In the first part of the list (Include on the List), we try to get 
respondents to think about and categorize what even they have little 
difficulty recognizing as their teachers. (For the most part,, their 
difficulties with these teachers were with how to categorize them according to 
our definitions, which is why we tried to provide more comprehensive 
definitions.) 

After this, we come along and remind respondents to include other 
teaching staff they may not have originally thought about, the teaching 
principals, etc. Also, we reworded the way we ask for this information. We 
tried not to repeat subtly different information in both the include and 
exclude list, especially the "who teach" and "who do not teach" phrases. 
Instead, we simply said to report Teaching Principals and the like in the 
"include" list, but we didn’t mention the corollary. On the one hand, we are 
afraid that if we tell respondents not to report principals and the like in 
the omit list, this information may be more readily seen and misunderstood by 
respondents. 'However, it should be noted that we are torn over this. We also 
are concerned that respondents may be confused by the fact that the omit list 
doesn’t inclusively cover all of the more common non-teaching staff, such as 
the principals, etc. However, in the end, we have decided to omit them in the 
belief that respondents are more likely to recognize that non-teaching 
principals are NOT teachers than they are to recognize that teaching 
principals ARE teachers. This is because in the latter case, the teaching 
position is often secondary to the other position, and therefore, more likely 
to be forgotten. Although redundant, we repeated the word teaching with each 
of the categories to clarify that all of the staff listed need to be teaching, 
not just the principals. 

Since many respondents had difficulty determining whether or not to 
report librarians and speech therapists, we included specific examples 
regarding them. We are not married to the wording of our examples, just to 
the notion that examples need to be provided that more explicitly help 
respondents understand how to report. 

Finally, respondents need a very specific definition of what we mean by 
part-time teaching, whether it be the one we used "who teach at least one 
class per week," or it be some other definition. This definition is given in 
the first entry under the Teaching Principals, etc., category. 

Now, because columns (f) through (o) are conceptually related to the 
instructions; because, next to the names, this is the information we are most 
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after; and because these columns take greater thought on the part of the 
respondents, we recommend that these columns be switched with columns (c) 
Race/Ethnicity, (d) New, and (e) Bilingual/ESL/ESOL. We also wonder if the 
Race/Ethnicity question should come last because it has the weakest link to 
teaching itself. 

We also strongly recommend that the questionnaire be redesigned to ask 
for information in a way that more closely resembles reality for respondents. 
As it is now, middle or combination schools have a lot of difficulty providing 
information in the format we ask for. It would be much better for them if we 
could break the data into three sections— Elementary School (K-5), Middle 
School (6-8), and Secondary School (9-12)— and collapse the data later to fit 
our needs. 

Even if the table can not be redesigned with three sections rather than 
two, still it is very important that each of the sections contain exactly the 
same categories, with two exceptions: the elementary section would not have 

the "vocational-education" section and the secondary school would not have a 
"general education" category. If the categories are duplicated, as we 
suggest, it may be necessary to widen the questionnaire using 8" x 14" paper 
because there will be 14 subject matter columns rather than the 10 that are on 
the form now. 

Another option would be to ask two separate questions. In the first, 
respondents would classify the teacher by grade range, and in the second, they 
would classify the teacher by subject matter. The advantage of this method is 
that it only requires 10 columns as opposed to 14 mentioned above. It would 
look something like this: 



Grade Range 



Subject Matter Taught 



Elementary K-6 


Secondary 7-12 


El em. 


Sp.Ed. 


Math 


Science. . . 


(G) 


(d) 


(e) 


(f) 


(9) 


(h) 



Other 

(i) 



In any event, the instructions that are now on page 3 need to be 
reworded. Under the first proposal above (asking for duplicate information), 
item 9 would become: 



9. How are columns (c) through (p) completed? 



The instructions that follow this question should be broken into three 
sections: (1) those that apply to elementary teachers, (2) those that apply to 
secondary teachers and (3) those that apply to both, so that respondents can 
easily find the section that applies to them. Also, many of the instructions, 
like the one telling respondents when to mark the "general elementary" column, 
can be eliminated because this information has been moved to the 
include/exclude list. 
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Looking back at the interviews, we learned that respondents either 
reported all of the subjects teachers were equally qualified to teach, or they 
agonized over making this decision. This leads us to propose two ways of 
asking respondents to report this information. As it is now, we tell 
respondents to choose the subject the teacher is most qualified to teach. One 
would not expect this to be difficult, but it is. One option, and the one we 
are recommending at this point, is to give respondents a decision rule they 
may be able to abide by: to just arbitrarily choose one of the subjects over 

the other(s). This is presented under the first option below. 

If further research demonstrates that respondents continue to mark more 
than one subject regardless of the decision rule we give them, then it will 
probably be necessary to go with their natural tendencies to report all of the 
subjects teachers are equally qualified to teach. Instructions appropriate 
for this tendency are given under the second option below. The advantage of 
this option is that respondents need not make any decisions about which 
subjects to report. We can always randomly choose one of the subjects after 
data capture. However, the success of this method depends on respondents 
recognizing that we want them to mark only one box if teachers teach more than 
one subject for which they are NOT equally qualified. Knowing how respondents 
want to report everything, this too may be a problem. 

Regardless of the option used, we suggest that the information be 
presented in typographical chunks, as done below. Typographically grouping 
the information may help respondents decipher the rather difficult, nested 
nature of these instructions. 



First Option: 

Elementary Teachers (K-6) 

If a teacher teaches only students in grade 6 or below: 

mark one and oniy one "X" under Elementary in the subject taught 
MOST by the teacher, 

unless the teacher teaches two or more elementary subjects 
EQUALLY, in which case arbitrarily choose one of the subjects over 
the other(s) and mark an "X" in it. 

Secondary Teachers (7-12) 

If a teacher teaches only students in grade 7 or above: 

mark one and only one "X" under Secondary in the subject taught 
MOST by the teacher, 

unless the teacher teaches two or more secondary subjects EQUALLY, 
in which case arbitrarily choose one of the subjects over the 
other(s) and mark an "X" in it. 

Teachers at Both Elementary (K-6) and Secondary (7-12) Levels 

If a teacher teaches both elementary and secondary students at 
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this school : 

mark one and only one "X" under either Elementary or Secondary in 
the subject the teacher teaches MOST, 

unless the teacher teaches two or more subjects EQUALLY, in which 
case arbitrarily choose one of the subjects over the other(s) and 
mark an "X" in it . 

Second option: 

Elementary Teachers (K-6) 

If a teacher teaches only students in grade 6 or below: 

mark one and only one "X" under Elementary in the subject taught 
MOST by the teacher, 

unless the teacher teaches two or more elementary subjects 
EQUALLY, in which case mark an "X" in all of the subjects the 
teacher teaches EQUALLY. 

Secondary Teachers (7-12) 

If a teacher teaches only students in grade 7 or above: 

mark one and only one "X" under Secondary in the subject taught 
most by the teacher, 

unless the teacher teaches two or more secondary subjects EQUALLY, 
in which case mark an "X" in all of the subjects the teacher 
teaches EQUALLY. 

Teachers at Both Elementary (K-6) and Secondary (7-12) Levels 

If a teacher teaches both elementary and secondary students: 

mark one and only one "X" under either Elementary or Secondary in 
the subject the teacher teaches MOST, 

unless the teacher teaches two or more subjects EQUALLY, in which 
case mark an "X" in all of the. subjects the teacher teaches 
EQUALLY. 



Month and Day School Closes 

As mentioned earlier, the question concerning the month and day school 
closes needs to be moved because it is conceptually unrelated to the listing 
activity. In its place we suggest providing an instruction that elucidates a 
question respondents often had about whether they were supposed to list the 
teachers alphabetically. This is in keeping with the notion of good 
information organization, for this is the point when respondents actually 
begin to write down the names. We suggest the following: 
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In the table below, list the teachers alphabetically, by grade, or 
by department — whatever is easiest for you. 

The instruction to report the month and day school closes is now in bold, 
while the rest of the table is not. We do not think that the listing 
instruction should be put in any bolder type than the column headings. 

Columns (a) and (b) — Line number and Name of Teacher 

The heading in column b should be made larger and it should tell 
respondents to print, since they questioned this. 

Print Last Name, then First Name of Teacher 

Column (c) — Race/Ethnicity 



Since we are suggesting that this column be moved to the right of the 
subject columns, we may want to re-label it "Teacher’s Race/Ethnicity" to 
clarify that we are talking about the teachers and not students, just in case 
respondents begin to think in terms of the students as a result of answering 
the subject columns. 

Column (d) — New Teachers 

This instruction should say Enter an "X" rather than Enter an "N." 



Columns (f) through (o) 

The current column heading "Mark (X) in ONLY ONE of columns (f) through 
(o)" is actually an instruction and not a descriptive heading. The heading 
should read "School Level and Subject Taught by Each Teacher." The "Mark (X) 
in Only One of columns (f) through (o)" should be placed beneath the column 
headings so that respondents don’t overlook it. This way their eyes should 
pass over the instruction when they move from the heading to where the the "X" 
goes. Perhaps this will help keep respondents from reporting more than one 

If ^ II 



Also, we suggest repeating the column titles (like Name, General 
elementary. . .Bil ingual /ESL/ESOL) at the top of the odd-numbered pages. We are 
not talking about the entire heading, just enough of the heading that 
respondents need not go back so far to stay properly aligned. 

VI. Conclusion 

We have made a number of recommendations that would substantially change 
the listing form’s present structure. We have based these recommendations on 
the results of cognitive research with the present form. However, when 
combined, our recommendations may change the questionnaire in ways we could 
not anticipate. Also, there are a number of places throughout the report in 
which we discuss the pros and cons of our recommendations. Take, for 
instance, the include/exclude instruction. We know respondents are havina 
difficulty accurately determining who to report as teachers. We also know 
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that respondents’ initial reactions to the Teacher Listing Form were that it 
looked burdensome. After filling it out, most respondents remarked that they 
did not think it was nearly as burdensome as they first thought it would be. 
Since respondents had so many problems correctly reporting their teachers, we 
made the include/exclude instruction more comprehensive, which has the 
unfortunate negative side effect of increasing its length some. Because of 
this and other similar complex questionnaire design issues in which we find 
ourselves needing to strike a delicate balance between helping respondents 
answer the questionnaire correctly, while at the same time continuing to 
motivate them to do so, we would be amiss if we did not end this report by 
saying that the recommendations we make should be tested before being 
implemented in an actual survey. 
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Table 1. Number of Interviews by Type of School 



Type of School 


Number Interviewed 


Publ ic 




Elementary (K-5) 


3 


Middle (6-8) 


3 


High (9-12) 


3 


Total 


9 


Private 1 




Combination 




1-7 


1 


K-8 


4 


8-12 


1 


K-12 


3 


High 




9-12 


1 


Total 


10 
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Table 2. Number of Schools by County 



County 


Number 


York, PA . 


1 


Frederick, MD 


5 


Spotsylvania, VA 
Richmond, VA 


4 


8 1 
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Teacher Listing Form 
for 

Public and Private Schools 

Schools and Staffing Survey 
1993-94 
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Note: If this school's grade range differs from the grade range printed above, 
please call the Census Bureau toll free at 1-800-851-2014. 
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Long-term substitute teachers (fill the role of a regular 
teacher for 4 weeks or more) 
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Month j Day 
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Bilingual/ESL/ESOL 
Mark (X) if applicable. 


Bilingual teachers use 
native language to 
varying degrees to 


instruct students with 
limited English 
proficiency. 

English as a Second 
Language (ESL) and 
English for Speakers of 
Other Languages (ESOL) 
teachers provide 
intensive instruction in 
English to students who 
have limited English 
proficiency. 

Note: Do not include 
regular foreign language 
teachers as Bilingual. 
ESL. or ESOL teachers. 
(e) 






















New 

Enter an m N m 
if the 

teacher's total 
years in the 
teaching 
profession at 
all schools 
and school 
districts (not 
just their 
current 

school/district } 
is less than 3 
years, not 
counting this 
school year . 
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Race/Ethnicity 

1 - American Indian or 
Alaska Native 

2- Asian or 
Pacific Islander 

3 - Hispanic 

(can be any race) 

4- Black 
(non-Hispanic) 

6-White 

(non-Hispanic) 

Enter a code for each 
teacher. 

(c) 
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94-01 

94-02 

94-03 

94-04 

94-05 

94-06 

94- 07 

95- 01 
95-02 
95-03 



Listing of NCES Working Papers to Date 



Title 



Schools and Staffing Survey (SASS) 
Papers Presented at Meetings of the 
American Statistical Association 

Generalized Variance Estimate for 
Schools and Staffing Survey (SASS) 

1991 Schools and Staffing Survey (SASS) 
Reinterview Response Variance Report 

The Accuracy of Teachers’ Self-reports 
on their Postsecondary Education: Teacher 
Transcript Study, Schools and Staffing 
Survey 

Cost-of-Education Differentials 
Across the States 

Six Papers on Teachers from the 
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